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I Graphical Interface Control System 

2 

3 Background of the Invention 

4 

5 The invention relates to a graphical user interface for use in a computer or data processing 

6 system which includes a principal display and one or more attached devices with 

7 secondary displays which are movable with respect to the first display. More specifically 

8 the invention relates to a graphical interface which permits a user to seamlessly control 

9 information display on multiple display devices with a single pointing device. 

H 

j| Description of the Prior Art 

la At the present time most desktop and laptop computer systems make use of a graphical 

11$ user interface wherein images on a display screen represent data and program icons on a 

Q virtual desktop. The user selects and controls the programs and data by use of a pointing 

S device, typically a mouse, trackball or touchscreen, which is programmed to move a cursor 

3 on the screen. The system differentiates between various input commands and takes 

17 actions using program logic which combines information derived from the pushing of 

18 keyboard and mouse buttons with information which reflects the position of the cursor on 

19 the display screen. 
20 

21 Personal data assistant devices (PDA's) generally also use some form of graphical user 

22 interface. More sophisticated devices, for example those using the Microsoft Windows CE 

23 operating system have interfaces and pointing devices which closely resemble those on 
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1 laptop computers. Less sophisticated PDA's, handheld video games and cellular 

2 telephones more typically accept inputs from a more simple set of directional keys and 

3 may display user information in the form of highlighted menu entries or simple lighted 

4 indicators or buttons. 
5 

6 Hardware and software is now readily available which allows devices in these and other 

7 categories (for example digital cameras, audio players etc.) to be docked or otherwise 

8 attached and/or interconnected to a computer system. The interconnected systems are, 

9 however, usually complicated and difficult for a user to control. It is not uncommon for a 
f i principal computer system to seize control of the inputs of a docked device, whereby the 
if keyboard and pointing device of the computer remotely control the docked device, but this 
I| usually is done in a manner which significantly changes the graphical interface on the 
jig docked device, and can be confusing for the user. The situation can become even more 
ill complicated when multiple devices are docked to a system at the same time and the user is 
13 required to sort-out the differences between the methods and interfaces used to control 
;jj6 them. 

17 

18 Summary of the Invention 

19 A graphical user interface for a computer system allows a user to select and control 

20 attached devices based upon the physical location of those devices relative to a principal 

21 display screen. The system determines the relative direction from the principal display 

22 screen toward each attached device and a graphical user interface allows the user to assume 

23 control of each such device by using a principal pointing device to move a cursor to the 
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1 edge of the display screen in the direction of the device. As the cursor moves to the edge of 

2 the principal display screen it seamlessly assumes control of the native display and 

3 graphical interface on the docked device. The user is thus able to interpret information 

4 from the docked device using a familiar interface and does not suffer from confusion about 

5 which device he is operating 
6 

7 Description of the Drawing 

8 The invention is described with reference to the drawings in which: 

9 Figure 1 schematically shows a computer system in accordance with the invention; and 

{§ Figure 2 schematically shows details of some displays on the screens of the principal CRT 

ft and a docked PDA. 

53 A Preferred Embodiment 

ill A system according to the invention comprises a desktop personal computer processor 10 

13 which is connected in a conventional manner with a CRT display screen 12, a mouse 

% pointing device 14, and a keyboard 16. In this embodiment the computer 10 is preferably 

17 programmed with the Microsoft Windows 98 operating system and is mainly controlled 

18 via a graphical user interface (GUI) which is displayed as the well known virtual desktop 

19 with program and data icons 101 and a cursor arrow 102. The cursor arrow moves on the 

20 screen 12 up, down, right and left in response to the user's movement of the mouse 14 on a 

21 table top. 
22 
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1 The system also comprises a number of separate processor devices which are removably 

2 interconnected with the computer 10 and which, by way of example, include a docked 

3 PDA 200, a digital camera 210, a laptop computer 220, a scanner 230, a cellular or 

4 cordless telephone 240 a second desktop computer and display 250, a television receiver 

5 260 and a VCR 270. The manner in which each of these devices is connected to the 

6 computer 10 is relatively unimportant and by way of illustration may include: a docking 

7 cradle 201 which supports the PDA on the right side of the cabinet of CRT display 12 and 

8 which couples to the PDA to the processor 10 via a multipin connector 202; a USB or 

9 other network bus connection 225, 226, 227 to the laptop computer 220, the scanner 230 
ji and the second desktop computer 250; infrared links 215a and 215b to the video camera 
tl 210, television receiver 260 and VCR 270 and a wireless radio link 245 to the cellular 
i| telephone 240. Each separate processor device includes its own visual display device. 
IS Again by way of example the laptop computer 220 and the PDA 200 respectively each 
13 include an LCD display screen (220a, 200a) and operate under control of sophisticated 
l| operating systems which support full virtual desktop displays with icons and cursors. In 
;i3f contrast the video camera 210 and cellular telephone 240 may comprise less sophisticated 

17 LCD screens (210a, 240a) which permit simple menu or icon selection via highlighting, 

18 but without full cursor display and control. The television receiver may have a relatively 

19 simple on-screen display 260a while the VCR 270 may have indicator lights 270a together 

20 with a numerical channel or time indicator 270b and the scanner 230 may simply have a 

21 display panel 230a with an array of indicator lights. Each of the separate processor devices 

22 will include its own native user interface software, as conventionally used in that category 

23 of appliance. 
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1 Sensors and other means are provided so that the computer 10 can be aware of the relative 

2 direction from the CRT display screen 12 to each of the separate processor devices. In the 

3 case of the PDA 200, the cradle 201 is shown permanently affixed to the right side of the 

4 screen and the direction from the screen to the PDA will thus be known to be to the right 

5 whenever the electrical interface though connector 202 is completed. Alternately, a switch 

6 203 or a noncontacting sensor may be used to sense when the PDA is present in the 

7 docking cradle. The directions from the display screen to the video camera 210 and to the 

8 television receiver/VCR (260/270) can be determined using an array of directional infrared 

9 sensors 216a, 216b and 216c and the direction to the cellular telephone can be determined 
S3 by using a directional antenna array 172 in the wireless interface. In general, the direction 
Iff from the CRT display to the larger hardwired devices, for example to the computers 220, 
il| 250 and scanner 230 will most easily be entered manually into the interface software data 
113, by the user when these devices are first connected or later moved. 

u 

|S During operation, the user selects to control a particular separate processor device by using 

;ji the mouse pointer 14 to move the cursor 102 to an edge of the display screen 12 in the 

17 relative direction of the selected device. Thus in the illustrations of Figures 1 and 2, 

18 moving the cursor to the top edge 310 of the screen 12 will select control of the video 

19 camera 210 , moving the cursor to the right center edge 300 of the screen will select 

20 control of the PDA 200, moving the cursor to the top left corner edge 340 will select the 

21 cellular telephone 240 etc. If two or more devices lie in the same general direction from the 

22 display screen, as in the case of PDA 200 and the second computer 250 ? then separate 

23 portions of the relevant screen edge can be assigned to select each device in a manner 
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1 which suggests their relative position to the user. For example, the portion of the right 

2 screen edge 300 which is used to select the PDA 200 can be chosen below the edge portion 

3 350 which is used to select the second computer 250 in order to give an impression that the 

4 second computer 250 is more distant from the screen 12 than is the PDA 200. In practice 

5 programmers skilled in the art will recognize that the cursor need only be moved into a hot 

6 zone near the edge of the screen to trigger the software to cause it to transfer control. 

7 

8 As the cursor 102 is moved to an edge to the screen, interface software in the computer 10 

9 checks to determine whether a controllable separate processor device lies in the indicated 
if direction and, if so, transfers the input signal from the mouse 16 to the pointer input of the 
;i| GUI software in that device. By way of example, in the Figures, moving the cursor to the 
i| top of the screen 310 will, seamlessly transfer the output of the mouse to the video 
jiyj camera's native interface and cause a highlighted entry to move to the bottom of a menu 
OS list on the camera LCD screen 210a. Further movement of the mouse in same direction 
M will then cause the highlighting to move up the menu list on the camera LCD screen, 
ft Similarly moving the cursor to the lower right edge 300 of the CRT screen 12 will transfer 

17 the mouse input to the native interface of the PDA and enable the user to move the cursor 

18 on the PDA screen . If the display on the selected separate processor device is a simple 

19 array of indicator lights, for example as illustrated on scanner 230, then movement of the 

20 mouse 14 may simply cause each light on the device to illuminate in sequence in response 

21 to the mouse movement. Similarly, if the display on the separate processor device is a 

22 numerical display, then the native GUI software may use movement of the mouse as a 

23 signal to increment or decrement the indicated function and numerical display reading. 
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1 

2 Depending on the characteristics of the native interfaces (for example in the laptop 

3 computer 220 or the PDA 200) in each device, it may also be appropriate to seamlessly 

4 transfer input signals from the computer keyboard 16 to a keyboard input for the separate 

5 processor the device. 
6 

7 It is preferred to blank-out the cursor 102 on the CRT display 12 whenever control is 

8 transferred to a separate processor device, which will give the impression that the cursor 

9 has actually moved across to the connected device. When possible, the cursor on the 
J! display of the connected device can have the same size and/or shape as the cursor on the 
ff screen 12 in order to further enhance the impression that a single cursor has actually 
|j moved across. Thus, as control is transferred to the PDA 200, a matching cursor 302 can 
ill be generated on the left edge 315 of the PDA screen 200a, directly opposite the point on 
ill the edge 300 of CRT screen 12 where the cursor 102 was last seen. It is also preferred to 
p place a warning icon at the appropriate edge of the display screen 12 to indicate that inputs 
ft are being directed to a remote device. The cursor display can also be animated to create a 

r j 

17 cartoon impression of movement while transfer of control is being effected. 
18 

19 Control inputs to the remote device can generally be discontinued whenever the cursor or 

20 other indication on the remote devices display moves to an edge adjacent the computer 

21 display 12, for example whenever the cursor 302 on the PDA display 200a moves back to 

22 its left edge 315. However, in some cases, depending on the nature of the display and 

23 native graphical interface in the separate processor device, it may be necessary for the user 
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to input a specific escape code via the mouse buttons or keyboard 16 to effect a return to 
the interface operating system in the computer 10. The techniques mentioned above can 
also be used to give the impression that a cursor is actually traveling back from the 
separate processor display to the CRT screen. 

It may be desirable to disable local controls of the connected processors whenever the 
pointing device 14 and/or keyboard 16 assume control thereof. Thus when control of the 
second desktop computer 250 is taken up by mouse 16, its own native keyboard and mouse 
will be rendered inoperative by the system software. 
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CLAIMS 



L A system for processing and displaying data comprising: 

a first processor which functions to generate an first image of a first graphical desktop user 
interface; 

a first display screen connected to the first processor to display the first image; 

a first pointing device connected to the first processor to control movement of a first cursor 

in the first image; 

second processor means which include second visual display means and which are 
movable in relation to the first display screen; 

means for communicating signals between the first processor and the second processor 
means; 

means which determine a first relative direction, from the first display screen to the second 
visual display means; and 

program means which expand the display of graphical desktop user interface onto the 
second visual display means at times when first processor is communicating with the 
second processor means and the second visual display means are also in the vicinity the 
first display screen so that movement of the first pointing device in the first relative 
direction causes the first cursor to move to and to appear to disappear off of an edge of 
first display screen and further causes the appearance of a new visual indication on the 
second visual display means. 
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1 2. The system of claim 1 wherein the second visual display means comprise a second 

2 display screen and wherein the program means function so that movement of the first 

3 pointing device in the first relative direction causes the first cursor to move to and 

4 disappear off an edge of the first display screen in a direction toward the second visual 

5 display means and to apparently seamlessly appear as a new cursor on the second display 

6 screen. 
7 

8 3. The system of claim 2 wherein the program means function so that the new cursor 

9 appears at an edge of the second display screen which is oriented toward the first display 
j i| screen. 

| 

IS 4. The system of claim 2 wherein the first pointing device further functions to control 

iia movement of the new cursor on the second display screen. 

ill 

S3 5. The system of claim 2 wherein the program means function to cause the first cursor to 

|| reappear on the first display screen whenever the new cursor is moved off the edge of the 

17 second display screen in a direction toward the first display screen. 
18 

19 6. The system of claim 1 wherein the first pointing device controls the appearance and 

20 apparent movement of the new visual indication on the second visual display means. 
21 

22 7. The system of claim 1 wherein the means for communicating is a docking cradle 

23 attached at an edge of the first display screen for supporting the second processor means. 
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8. The system of claim 7 wherein the means which determine comprise means which 
sense that the second processor means are in the cradle. 

9. The system of claim 1 wherein the means for communicating are a wireless interface 
and wherein the means which determine comprise a directional antenna array. 

10. The system of claim 1 wherein the means which communicate are an infrared light 
interface and the means which determine are directional infrared sensors. 

11. The system of claim 1 wherein the second visual display means comprise one or more 
indicator lights. 

12. The system of claim 1, wherein the second processor means is a device selected from 
the group consisting of: personal data assistants, desktop personal computers, laptop 
computers, digital cameras, audio players, video games, cordless telephones, cellular 
telephones, television receivers, VCR's and scanners. 

13. A method for processing and displaying data comprising: 

generating a first graphical desktop user interface image on a first display screen with a 
first processor; 

using a first pointing device to control movement of a cursor on the first interface image; 
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1 communicating signals between the first processor and second processor means, which 

2 second processor means are movable in relation to the first display screen and which 

3 include second visual display means; 

4 determining a first relative direction from the first display screen to the second visual 

5 display means; and 

6 expanding the display of graphical desktop user interface onto the visual display means at 

7 times when first processor is communicating with the second processor means and the 

8 second processor means are also in the vicinity the display screen so that movement of 

9 the first pointing device in the first direction causes the cursor to move to and to appear to 
11 disappear off of an edge of first display screen and further causes the appearance of a new 
jil visual indication on the second visual display means. 

1J1 14. A system for processing and displaying data comprising: 

is 

0 a first processor which functions to generate an first image of a desktop graphical user 

Il3 interface; 

a first display screen connected to the first processor to display the first image; 

17 a first pointing device connected to the first processor to control movement of a cursor in 

18 the first image; 

19 second processor means which include second visual display means and which are 

20 movable in relation to the first display screen; 

21 means for communicating signals between the first processor and the second processor 

22 means; 
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1 means which determine a first relative direction from the first display screen to the second 

2 visual display means; and 

3 program means operating within at least the first processor to allow selection or movement 

4 of an indication on the second visual display means in response to operation of the first 

5 pointing device at times when first processor is communicating with the second processor 

6 means and the second processor means are also in the vicinity the first display screen so 

7 that operation of the first pointing device in the first direction causes the cursor to move to 

8 and to appear to disappear off of an edge of first display screen together with a 

9 corresponding visual indication on the second visual display means. 

H 



CH2000A Carl Herman Haken 



Page 13 of 14 



Graphical Interface Control System 

Abstract 



A graphical user interface for a computer system with attached processor devices transfers 
input control for a pointing device on a principal computer display to native display 
devices and interfaces on attached devices whenever the system senses that a display 
cursor has moved to the edge of a display in the direction of the attached device. 
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As a named inventor, ! hereby appoint the following registered practiti oners) to prosecute this applic ation and to trans act al i business in the Patent 
and Trademark Office connected therewith: Q Customer Number 

OR 



□ Registered practitioners) name/registration number listed below 



Place Customer 
Number Bar Code 
/ ahff/ hftrR 



Name 


Registration 
Number 


Name 


Registration 
Number 


none 









Ariditinnah^ sheet PTO/SB/02C attached her eto 



Direct all correspondence to: □ Customer Number 

or Bar Code Label 



OR Q Correspondence address beiow 



Name 



Carl Haken 



Address 



5 Old Neversink Road 



City 


Danbury 


State 


CT 


ZIP 


06811 


Country 


USA 


Telephone 


203 792 5837 


Fax 


203 748 3549 



I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information and b ehef are 
believed to be true; and further that these statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under 18 U.S.C. 1001 and that such willful false statements may jeopardize the vai idity of the 
application or any patent issued thereon. 



Name of Sole or First Inventor: 



D A petition has been filed for this unsigned inventor 



Given Name (first and middle Nf anvl) 



Family Nam.fi or Surname. 



Carl Herman 



Haken 



Inventor's 
Signature 



Date 



3/6/00 



Residence: City 



Danbury 



State 



CT 



Country 



USA 



Citizenship 



USA 



Post Office Address 



5 Old Neversink Road 



Post Office Address 



City 



Danbury 



State 



CT 



ZIP 



06811 



Country 



USA 



O Additional inventors are being named on the supplemental Additional inventor(s) sheet(s) PTO/SB/02A attached hereto 
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